The Color-to-Gray (C2G) image quality evaluation algorithm based on structural similarity does not make full use of the gradient feature of the image, and the contrast similarity feature ignores the consistency of the continuous color blocks of the image, thus leading to a large difference between the algorithm and the subjective judgment of human vision. A C2G image quality evaluation algorithm named C2G Visual Similarity Index Measurement (C2G-VSIM) was proposed based on Human Visual System (HVS). In this algorithm, the color image was regarded as the reference image, the corresponding decolorization image obtained by different algorithms was regarded as the test image. By applying color space conversion and Gaussian filtering to these reference and test images, taking full account of the characteristics of image brightness similarity and structual similarity, a new color consistency contrast feature was introduced to help C2G-VSIM to capture the global color contrast feature; then the gradient amplitude feature was also introduced into C2G-VSIM to improve the sensitivity of the algorithm to the image gradient feature. Finally, by combining those above features, a new imgage quality evaluation operator named C2G-VSIM was obtained. Experimental results on Cadík's dataset showed that in terms of accuracy and preference evaluation, the Spearman Rank Order Correlation Coefficient (SROCC) between C2G-VSIM and subjective assessment of human visuality was 0.8155 and 0.7634, respectively, the accuracy was improved significantly without increasing the time consuming compared to C2G Structure Similarity Index Measurement (C2G-SSIM). The proposed algorithm has high consistency compared to human visuality, as well as simple calculation, which can effectively and automatically evaluate decolorization images in actual project with large scale.